System and method for improved optical character recognition for automated set-top box testing

ABSTRACT

The present application provides a user configurable test system for set-top boxes (STB) and other consumer devices providing video output. In particular, it provides for a method of improving an Optical Character Recognition (OCR) process in such test systems.

FIELD

The present application pertains generally to the automated testing ofset-top boxes (STB) and other audio visual equipment.

BACKGROUND

A set-top box (STB) also known as a digibox or set-top unit (STU) is adevice that connects to a television and an external source of signal,turning the signal into content which may then be delivered as anAudiovisual (A/V) signal for display on the television screen or otherA/V device. Most frequently, the external source of signal is providedby a satellite or cable connection.

As with other consumer products, both manufacturers and suppliers arekeen to ensure that products operate correctly and as specified.Initially and to this day, a significant part of the testing isperformed manually, whereby a tester issues a command to the STB whichmay be via the user interface on the STB itself or via a remote controldevice as illustrated in FIG. 1, and observes the response of the STB ona TV display. As is shown in FIG. 1, a typical STB 10 has a number ofsignal inputs including an RF signal 14, which may for example be from asatellite or cable connection. An A/V signal 16 may also be provided asinput allowing the set-top box to feed through a signal to a televisionfrom a VCR, DVD, Blu-ray disc, Media Juke Box or other similar device.The output to the television is an A/V signal 18 which may be providedover a variety of standard interfaces including SCART and HDMI. To allowthe user control the operation of the STB, a number of buttons andsimilar controls may be provided on the STB itself. Additionally andmore commonly employed by users, the STB may have an infra red (IR) orwireless remote control input configured to operate with a remotecontrol device 12.

As manual testing can be time consuming, prone to error and in someinstances lacking accuracy, an effort has been made to automate some ofthese tests. In respect of these automated tests, it will be appreciatedthat this testing is typically performed on the end product by userswithout necessarily any detailed knowledge or access to the internalcircuitry of the STB. Accordingly, the testing of STB's is generallyperformed on a “black-box” basis, where only the inputs and outputs areavailable for modification and examination. Accordingly test methods andsystems have been developed specifically for set top boxes, an exampleof such a system is StormTest™ provided by the S3 Group of Dublin,Ireland which may be employed to test STBs, televisions, and similardevices such as digital media players. An arrangement for a STB testsystem is shown in FIG. 2. The STB test system 20 comprises a controller28 which manages the test function and interacts with the other featuresof the STB test system. The other features include an output interfacefor controlling a remote control device 12, allowing commands to be sentto the STB and an input interface for receiving the video and/or audiosignals from the STB 10. This input interface may include an audiocapture device 24 for accepting the audio as a test signal and/or aframe grabber 22 or similar device for accepting the video frames fromthe STB. The data captured is then made available to a processor foranalysis, which in turn produces a test result and provides this to auser. Thus during a typical test, the Test System issues a command orsequence of commands to the STB, suitably via the remote controlinterface. Each frame of video and/or the audio is captured and made toavailable to the Test System for analysis of the STB response to thecommands issued to it.

Typically, tests might include the generation of a “change channel”command to the STB followed by analysis of the audio and/or videooutputs to ensure that the channel change command was correctly receivedand executed by the STB.

Set-top boxes are complex devices containing a main powerful embeddedCPU and various peripheral devices. They generally run a sophisticatedoperating system (e.g. Linux, or VxWorks) and perform complex functions,and run a large and complex software stack on that operating system.These devices generally present to the users a sophisticated graphicaluser interface with on-screen menus and graphics. Automation of thetesting typically involves, amongst other areas, optical characterrecognition (OCR) to read text from the screen and compare it with knownor expected value, to determine if the user interface presented to theuser is as expected.

As shown in FIG. 2, OCR may be performed on a captured frame or asection of a frame. Typically, in configuring a test routine, the personconfiguring the test will identify the section of the frame where textis to be found. The person may also configure the test by defining theexpected result from the OCR process. The expected result may be apre-determined known value or it may obtained elsewhere in the testroutine. Additionally, the OCR process may be employed to obtain andstore information from the device under test. OCR is generally performedby a software component known as an “OCR Engine”. There are several OCRengines (software product) generally available for this purpose, i.e.the engines can process images and extract text from them. However,there are several practical difficulties in using an OCR engine in thecontext of a frame captured from a STB or device with a display, e.g. adigital television. One of the reasons for this is that OCR enginesconventionally come from the document processing area, which has severaldifferences to images captured from a video stream. The resolution ofdocument scanners used to capture document images is generally muchgreater than the resolution of images, even high definition images,captured from video.

Documents generally have a fixed foreground/background contrast pattern:generally dark text on a light background. In addition, the contrast fora given document page is not variable. Once the OCR engine hasdetermined the foreground and background colours for a page, this willnot change. Thus for example it is known to use filters to optimise anOCR engine, for example where the page colour is not white. Howevercontrast on modern dynamic user interfaces can be highly variable. As anexample, transparent panes in the user interface are commonplace andthus the contrast of the text will vary as the background televisionprogram changes. To make matters worse, images captured from videostreams can be quite noisy. As a result, the present inventor'sexperience has been that the accuracy of a typical OCR engine whenemployed with captured video may only be in the region of 65-90%.

It would be beneficial to improve the performance of OCR engines inanalysing video images.

SUMMARY

In particular, the present application provides systems and methods inaccordance with the claims which follow.

DESCRIPTION OF DRAWINGS

The present application will now be described with reference to theaccompanying drawings in which:

FIG. 1 is an illustration of an exemplary STB known in the art;

FIG. 2 is an illustration of a conventional prior art STB test systemhaving OCR;

FIG. 3 is a block diagram of an aspect of STB test system according toan embodiment of the present application;

FIG. 4 illustrates an exemplary arrangement for selecting a filterconfiguration according to another embodiment; and

FIG. 5 illustrates a method for use in the arrangement of FIG. 4.

DETAILED DESCRIPTION

The present application is based on the premise that developing an OCRengine specifically for captured video images may be a costly and timeconsuming process. Instead, it is desirable to provide a method thatimproves the performance of existing OCR engines, for example which areintended for scanned documents.

The present application improves the performance of OCR engines oncaptured video frames or sections thereof. The improvement is obtainedby pre-processing the image (the captured frame or a part thereof)before submission to the OCR engine. In particular, it has been found bythe present inventor that by processing an image with an image filter,the performance of the OCR engine may be improved. The difficulty isthat whilst certain image filters may work perfectly in certainsituations they may result in worse performance in others.

An exemplary test system for testing a STB may generally employ theknown STB test system of FIG. 2 and thus include a first interface forcontrolling the operation of a set-top box, for example by sendingcommands to the STB via an IR remote control. Although, it will beappreciated that other control inputs/interfaces may be employed forexample a direct serial connection may be employed if the STB has suchan interface available. It will be appreciated that an interfaceemployed to control the STB is commonly employed in conventional testsystems for STB's and thus their design and operation would be readilyunderstood and familiar to those skilled in the art. The test system isconfigured to analyse video from the STB along with other outputsincluding for example audio.

Specifically a second interface is employed to acquire one or moreoutputs from the STB. This second interface may include a frame grabberfor capturing the video frames from the STB and/or an analog to digitalconverter for capturing the audio output from the STB. It will beappreciated that the technology associated with these elements wouldalso be familiar to those skilled in the art. However, suitably theframe grabber is synchronised to the video frame timing allowing it tocapture one complete video frame at a time. It will be appreciated thatwhere a digital output is available from the set-top box, therequirement for an analog frame grabber/audio capture device may beobviated and replaced by a digital interface which directly captures thevideo frames being transmitted. Where the device under test has anintegrated display for displaying the video, the video may be capturedusing a video camera directed at the display. As with existing systems,the system may select a particular region of interest from the frame.The region of interest would typically be pre-defined during theconfiguration\setting up process of a test routine.

In the exemplary embodiment illustrated in FIG. 3 it may be seen thatthe frame capture and OCR engine\processes are unchanged and instead animage processor (filter) 40 is provided to pre-filter the image (whichas explained above may either be the entire captured frame or morelikely a pre-selected section of the captured frame) before processingby the OCR engine to extract text from the image.

The filter of the image processor 40 is a configurable filter such thatthe filter function applied to the image may be varied by the system.More specifically, the configuration for the filter for an image at aparticular point in a test routine is suitably pre-set during an initialset-up configuration process where the test routine performed by thetest system is prepared.

The mode of operation will now be explained with reference to the methodof determination of a configuration for the configurable filter as shownin FIG. 4 in which the configurable filter 40 comprises a sequence ofdifferent image filters 42 a-f which may be applied sequentially to animage 44 from a captured frame or part thereof. As will be appreciatedfrom the explanation which follows, each of the filters in the sequencemay or may not be used to filter the image. Thus, the overall filter maybe configured by the selection of which particular filters to use.Additionally, each filter may be configurable by one or more parameters(shown by the numerals 0-5 for each filter), which adjust the filtercharacteristics. One of these parameters, e.g. 0 may indicate that thefilter is not be used. After filtering has been completed, the filteredimage is passed to the OCR engine 24 for processing, where therecognised text may be compared with an expected result to determine thesuccess or not of a particular test. In certain circumstances, there maynot be an expected result and in this case, the result may simply bestored for reference or use subsequently in the test routine. This sameroutine may also be employed in the process to determine the filterconfiguration as described in greater detail below.

As examples, the following image filters have been determined to improvethe accurate detection of text by the OCR engine when operating oncolour images:

-   -   Selectively removing one or more colour components from the        original image, with or without conversion to greyscale of the        resulting image.    -   Adjusting image contrast    -   Inverting the colours in the image    -   Blurring the image    -   Sharpening the image    -   Zooming the image so that it is scaled up, interpolating pixels        from the original image to create the new image.

It will be appreciated that whilst each of these may be regarded as animage filter. It will be appreciated that a general parameter for afilter may be whether it is employed or not. A specific parameter in thecase of removing one or more colour components would be the colourcomponents to be removed, thus in the case of a RGB (Red Green Blue)image, the filter configurations might be removal of:

-   -   a) R    -   b) G    -   c) B    -   d) RG    -   e) RB    -   f) GB

In the case of adjusting image contrast, an individual parameter mightbe whether to increase or decrease the contrast or the amount ofcontrast adjustment. Similarly, in the case of blurring or sharpeningthe degree of blurring or sharpening would be an individual filterparameter. In the case of an image filter for “zooming” the image sothat it is scaled up, individual parameters may be the degree of scalingand/or the selection of a particular type of interpolation e.g.bi-cubic, linear etc.

An exemplary method of selecting a configuration for the filter is shownin FIG. 5. In this exemplary method a user when configuring the testsystem, and as would be performed conventionally, may optionallyidentify a section of the screen (captured frame) to be analysed. Theuser specifies the expected result of the OCR process. The expectedresult is used to compares the output\OCR result of different filterconfigurations on a sequence of images to determine the performance ofthe different filter configurations. Each of the sequence of imagescontains substantially the same text content, i.e. a user visually seesthe same text in each image of the sequence. For example, capturing 100sequential frames of video having an OSD (On Screen Display) box shownwould have the same text content visible to a user in the box in each ofthe frames on the screen. It will be appreciated however that a bit forbit comparison of the captured frames may well show them to becompletely different for the reasons explained above including noise andpossibly changing video content behind the OSD box. Suitably, a sequenceof images are acquired as an initial step and the same sequence ofimages are used for comparing the performance of each filterconfiguration as the on-screen content may time out after a few secondsand an OSD disappear altogether. It will be appreciated that thesequence of images may be obtained as part of this process or they mayhave been obtained separately and stored.

A first filter configuration is selected 52 to be used to filter theselected 54 first image in the sequence. In the exemplary arrangement ofFIG. 4, a series of filters are applied in a chain to the image. Thefirst filter configuration may have a “null” setting for each filterwhich does not alter the image. Pictorially, in FIG. 4, this isrepresented by the setting “0” for a filter. This first filterconfiguration is used to set the filter and the first image is processed56 by the filter using this configuration to provide a filtered image.The filtered image is then passed 58 through the OCR engine and theextracted text compared 60 with the expected result to determine theperformance of the filter. It will be appreciated that the expectedresult need not be the entire of the extracted text but may in fact be apart of the text. For example, the person configuring the test maysimply require the presence of a particular word in the extracted textrather than an exact match to the whole of the extracted text. Adecision 62 is then made based on the outcome, where the filterperformance is negative (i.e. text not recognised correctly), then thenext filter configuration may be selected 52 and the test repeatedagain. Where the filter performance is determined to be positive, thetest may be repeated for the next image in the sequence 54. Where theperformance of a particular filter configuration is positive for allimages in the sequence, then that configuration may be selected andstored 64 within the test routine for future use in a test process. Atwhich point the configuration process for that aspect of the test iscompleted.

Alternatively, all of the filters configurations may be tested with theaccuracy of each determined (i.e. in how many of the sequence of imagesdid a filter result in the OCR producing the expected result) and thefilter with the best accuracy selected.

Using this method, it is possible to choose a filter configuration whichresults in a correct match for the recognised text.

Whilst the above description refers to a sequence of images havingsubstantially the same text content, it will be appreciated that thesequence may not be the actual sequence of frames and the training set(sequence of images) may be for example be captured frames chosen atintervals apart.

It will be appreciated that where the method is configured to select thefirst filter configuration that results in 100% accuracy that the filterconfigurations to be tested may be chosen in random order viatraditional Monte Carlo methods, thus avoiding locally suboptimal blocksof filters if filters are tried in strict order of definition.

In another variation, it is also possible to generate the training setfrom the live signal, by capturing images “live” and accumulating thesecaptures images on local storage. As long as each new image whicharrives is recognised correctly, there is no need to re-train with allexisting captured images (since these, by definition, will also havematched already). However once an image is captured which does not matchthe expected text, the captured images form the training set, and thealgorithm starts the search for a better filter using the capturedimages. Where a further image does not match the expected result with afilter, it may be added to the training set until a suitable trainingset has been selected.

This training may be time limited (so that if all images match theexpected text for a user-defined time period, the current best filter(configuration) is judged good enough and saved).

Given an image or set of images, it is possible for the system toautomatically determine with a high degree of accuracy the expected textwithout user input. This is based on the observation that whileincorrectly recognised text is typically random, the correctlyrecognised text is always the same. Therefore the system can “guess”that the string which appears most frequently in the recognition resultsis likely to be the text that the user wanted, and in the majority ofcases it will be correct. Thus although the above method refers to theuser entering an “expected” text result, it may not be necessary for theuser to do this. In an alternative variation, the extracted text from afirst image in a sequence may be employed as the “expected” result forthe remainder of the images in the sequence, i.e. consistency of resultis determined to equate to accuracy of result. It is possible of coursethat the OCR process may consistently fail to recognise the textcorrectly and that this alternative variation may not be appropriate.

Thus is it possible, without user intervention of any kind, to determinea set of image processing filters to apply to the captured image whichwill improve the accuracy of recognition.

It will be appreciated that whilst the present application has beendescribed generally with respect to testing of STBs, it is equallyapplicable to testing other devices such as digital televisions, digitalmedia players, DVD players and consumer devices such as mobile phonesand PDA's. It will be appreciated that whilst digital televisions,digital media players and DVD players may have a remote control (e.g.IR) input for receiving test commands from the test system, otherdevices may require a different interface in the test system for sendingtest commands to the device under test.

Moreover, it will be appreciated that the presently described techniquesmay also be employed directly without a test configuration process. Inparticular, whilst the above method has been described with respect tousing an initial configuration routine to establish\store the correctfilter parameters for performing a particular test. The method may alsobe used in a live scenario to determine text content from a sequence ofcaptured images. In such an arrangement, a sequence of captured frames(or parts thereof) may be passed through the configurable filter using afirst filter setting and then through the OCR engine to provide a textresult. Where the text result is consistent for all (or a significantproportion thereof) of the captured frames, the text result may beregarded as valid. Where the text is not consistent, the process may berepeated with a different filter configuration. This process may berepeated, varying the configuration each time, until a valid result isdetermined. It will be appreciated that this process may be of usegenerally to video captured from a device under test and may be employedgenerally to identify text in video content.

The invention claimed is:
 1. A test system comprising: a processor to:obtain a video frame, comprising images in sequence, from a device undertest; select a filter configuration for at least one filter to test;filter an image included in the images in sequence using the filterconfiguration to generate a filtered image; extract text from thefiltered image using an optical character recognition (OCR) engine;determine a filter performance for the image based on comparing the textextracted from the filtered image to an expected result; when the filterperformance is a positive result, filter another image included in theimages in sequence using the filter configuration; when the filterperformance is a negative result, repeatedly select a different filterconfiguration and filter the image using the different filterconfiguration until the positive result is determined; and determine asuitable filter configuration when the filter performance for each imagein the images in sequence is positive for a same filter configuration.2. A test system according to claim 1, where the at least one filter isadapted to perform one or more of the following on the image:selectively remove one or more color components adjust image contrast;invert colors; blur; sharpen the image; or zoom the image so that theimage is scaled up using interpolation.
 3. A test system according toclaim 1, where the at least one filter includes a plurality of differentfilters and the filter configuration indicates which of the plurality ofdifferent filters are to be used to filter the image.
 4. A test systemaccording to claim 1, where each of the at least one filter has aplurality of different filter settings, and where the filterconfiguration defines the plurality of different filter settings to beemployed for filtering the image.
 5. A test system according to claim 1,where the expected result is determined by optical characterrecognition.
 6. A test system according to claim 1, where the processor,when obtaining the video frame, is to: capture the video frame using aframe grabber.
 7. A test system according to claim 1, wherein the deviceunder test is one of: a set-top box; a digital media player; or a DVDplayer.
 8. A test system according to claim 1, where the processor, whenobtaining the video frame, is to: acquire the video frame from displayof the device under test using a camera.
 9. A test system according toclaim 8, where the device under test is one of: a television; a monitor;a mobile telephone; a personal digital assistant; a digital camera; or alaptop.
 10. A method comprising: providing, by a device, a sequence ofimages of substantially a same content; selecting, by the device, afilter configuration for a filter for pre-processing an image of thesequence of images, from a plurality of available filter configurations;pre-processing, by the device, the image using the filter configurationto generate a pre-processed image; submitting, by the device, thepre-processed image through optical character recognition (OCR) toextract text from the pre-processed image; determining, by the device, afilter performance for the image based on comparing the text extractedfrom the pre-processed image to an expected text; when the filterperformance is a positive result, pre-processing, by the device, anotherimage included in the sequence of images using the filter configuration;when the filter performance is a negative result, repeatedly selecting,by the device, a different filter configuration and pre-processing theimage using the different filter configuration until the positive resultis determined; and determining, by the device, a suitable filterconfiguration when the filter performance for each image in the sequenceof images is positive for a same filter configuration.
 11. A methodaccording to claim 10, further comprising: pre-processing eachsuccessive image in the sequence of images and performing the OCR oneach successive image that is pre-processed as long as the textextracted from a preceding image is determined to be correct.
 12. Amethod according to claim 10, further comprising: pre-processing eachimage in the sequence of images; performing the OCR on all of the imagesin the sequence that are pre-processed; and where determining thesuitable filter configuration includes determining whether the filterconfiguration is suitable when the text extracted from each of theimages in sequence is determined to be correct.
 13. A method accordingto claim 10, where the filter configuration comprises one or more filtertypes selected from a plurality of different filter types.
 14. A methodaccording to claim 10, where the filter configuration comprises one ormore parameters for the filter.
 15. A method according to claim 10,where the filter comprises a plurality of sub-filters which are appliedin combination, and the filter configuration comprises one or moreparameters for each of the plurality of sub-filters.
 16. Anon-transitory computer-readable storage medium storing instructions,the instructions comprising: one or more instructions which, whenexecuted by a processor, cause the processor to: obtain a video frame,comprising images in sequence, from a device under test; select a filterconfiguration for at least one filter to test; filter an image includedin the images in sequence using the filter configuration to generate afiltered image; extract text from the filtered image using an opticalcharacter recognition (OCR) engine; determine a filter performance forthe image based on comparing the text extracted from the filtered imageto an expected result; when the filter performance is a positive result,filter another image included in the images in sequence using the filterconfiguration; when the filter performance is a negative result,repeatedly select a different filter configuration and filter the imageusing the different filter configuration until the positive result isdetermined; and determine a suitable filter configuration when thefilter performance for each image in the images in sequence is positivefor a same filter configuration.
 17. A non-transitory computer-readablestorage medium according to claim 16, where the at least one filter isadapted to perform one or more of the following on the image:selectively remove one or more color components; adjust image contrast;invert colors; blur; sharpen the image; or zoom the image so that theimage is scaled up using interpolation.
 18. A non-transitorycomputer-readable storage medium according to claim 16, where the one ormore instructions further cause the processor to: transmit commands tothe device under test via a remote control interface.
 19. A test systemof claim 1, further comprising: a remote control interface to transmitcommands to the device under test.